Depth with nonlinearity creates no bad local minima in ResNets
نویسندگان
چکیده
منابع مشابه
Depth Creates No Bad Local Minima
In deep learning, depth, as well as nonlinearity, create non-convex loss surfaces. Then, does depth alone create bad local minima? In this paper, we prove that without nonlinearity, depth alone does not create bad local minima, although it induces non-convex loss surface. Using this insight, we greatly simplify a recently proposed proof to show that all of the local minima of feedforward deep l...
متن کاملNo bad local minima: Data independent training error guarantees for multilayer neural networks
We use smoothed analysis techniques to provide guarantees on the training loss of Multilayer Neural Networks (MNNs) at differentiable local minima. Specifically, we examine MNNs with piecewise linear activation functions, quadratic loss and a single output, under mild over-parametrization. We prove that for a MNN with one hidden layer, the training error is zero at every differentiable local mi...
متن کاملLocal Minima Embedding
Dimensionality reduction is a commonly used step in many algorithms for visualization, classification, clustering and modeling. Most dimensionality reduction algorithms find a low dimensional embedding that preserves the structure of high-dimensional data points. This paper proposes Local Minima Embedding (LME), a technique to find a lowdimensional embedding that preserves the local minima stru...
متن کاملSimple Gauge-mediated Models with Local Minima
We describe a simple class of supersymmetric gauge theories that can act as supersymmetry-breaking sectors for gauge-mediated supersymmetry breaking. The models have a local supersymmetry-breaking minimum along a direction in field space where a singlet gets a large expectation value. The potential along this direction has a runaway behavior stabilized by supersymmetry breaking in the effective...
متن کاملContinuity of Minima: Local Results
This paper compares and generalizes Berge’s maximum theorem for noncompact image sets established in Feinberg, Kasyanov and Voorneveld [5] and the local maximum theorem established in Bonnans and Shapiro [3, Proposition 4.4].
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: Neural Networks
سال: 2019
ISSN: 0893-6080
DOI: 10.1016/j.neunet.2019.06.009